The Latest From Pipeline: Your Data Engineering Resource Hi past, present or future data professional! I hope you’ll indulge my sharing of an important career milestone; unfortunately, I’m not retiring with a gold Rolex snug on my wrist. Instead, this week marks 3 years in data engineering. I’m excited about this work-iversary because it marks a period of time in which I’ve been working in data engineering longer than my prior non-data job. If you’ve read my work you know I’ve taken an...
3 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline: Your Data Engineering Resource Hi past, present or future data professional! Browsing through files recently, I found 100+ old resumes I used to apply for data jobs in 2021. While data science is sold as a “good career”, the truth is it’s always been tough to break in. Those looking for jobs need to do more than ever to distinguish themselves. For anyone looking for a job you may have been taught to network with recruiters and hiring...
8 days ago • 1 min read
The Latest From Pipeline: Your Data Engineering Resource Hi past, present or future data professional! One of the worst habits you can adapt in data engineering is to accept a tool or technology as a black box you’ll simply never understand. For me, for the longest time, this was virtual machines. I would commit and troubleshoot code that would run within a VM but never quite knew how to interact with it. Recently, I provisioned a Google Cloud Compute Engine instance for a junior teammate and...
10 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline: Your Data Engineering Resource Hi past, present or future data professional! I bet when you think of data engineering (or software engineering, for that matter) you don’t think about development pace. After all, the buzzworthy anecdotes and lofty promises surround performance of a build, not the speed of its architects. But adaptability and the ability to work very quickly at times is a role expectation you won’t see listed on a job...
16 days ago • 1 min read
The Latest From Pipeline: Your Data Engineering Resource Hi past, present or future data professional! Have you heard about the programmer who successfully automated his job to the point where he did no work for 5 years? Unfortunately, automation efforts are rarely this comprehensive—or successful. In fact, spending time on automation often results in a time sink, requiring more time to conceive and create the automation than time it saves you. In the latest from Pipeline, I provide a simple...
18 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline: Your Data Engineering Resource Hi past, present or future data professional! For many, perhaps even yourself, it’s the most dreaded time of the year: Back to school. While data engineering doesn’t require “supplies”, it can be helpful to pick up a book or two. Aside from getting industry context, your latest data read can provide a helpful talking point in interviews or even impress in casual discussion. Though not quite a textbook, I...
21 days ago • 1 min read
The Latest From Pipeline: Your Data Engineering Resource Hi past, present or future data professional! Have you gotten a bit too comfortable with your 2-week sprint grind? In a professional environment, data engineers and other software developers enjoy breathing room to freely code, test and deploy products. But what about those times where your data engineering task is scoped out, assigned and expected to be completed on the same day? In the latest from Pipeline you’ll learn what a data...
23 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline: Your Data Engineering Resource Hi past, present or future data professional! Few aspects of data engineering are as shame-inducing as saying, after a failed deployment, “But it ran in my environment!” In my first year as a data engineer I was that guy who made excuses like this and grew frustrated that I would complete a build and then struggle to push it over the finish line. Here’s what helped me: Learning the subtle but important...
30 days ago • 1 min read
Extract. Transform. Read. A newsletter from Pipeline: Your Data Engineering Resource Hi past, present or future data professional! Data engineering can be dangerous; ok—not, like, physically, but by building and maintaining data infrastructure, data engineers are given a surprising amount of access and responsibility. Every commit, table alteration and deletion must be made with care. It took 2 years, but I finally learned a shortcut to make developing SQL staging tables less risky and more...
about 1 month ago • 1 min read